Policy Analysis with Incredible Certitude

Analyses of public policy regularly express certitude about the consequences of alternative policy choices. Yet policy predictions often are fragile, with conclusions resting on critical unsupported assumptions or leaps of logic. Then the certitude of policy analysis is not credible. I develop a typology of incredible analytical practices and gives illustrative cases. I call these practices conventional certitude, dueling certitudes, conflating science and advocacy, wishful extrapolation, illogical certitude, and media overreach. Acknowledgments: This paper was presented in February 2011 as a Leverhulme Lecture at the Institute for Fiscal Studies, in the framework of my Leverhulme Visiting Professorship at University College London. The research was supported in part by National Science Foundation grant SES‐ 0911181. I am grateful for comments from Teresa Delgado, Joel Horowitz, Richard Lempert, Francesca Molinari, John Mullahy, Daniel Nagin, James Poterba, and two referees. I have also benefited from the opportunity to present parts of this work in seminars at NYU, the Congressional Budget Office, and the University of Essex. Analyses of public policy regularly express certitude about the consequences of alternative policy choices. Point predictions are common and expressions of uncertainty are rare. Yet policy predictions often are fragile. Conclusions may rest on critical unsupported assumptions or on leaps of logic. Then the certitude of policy analysis is not credible. In a research program that began with Manski (1990, 1995) and continues to develop, I have studied identification problems that limit our ability to credibly predict policy outcomes. I have argued that analysts should acknowledge ambiguity rather than feign certitude, expressing partial knowledge by reporting interval rather than point predictions of policy outcomes. And I have shown how elementary principles of decision theory may be used to make reasonable policy choices using such interval predictions. Manski (2007) exposits core ideas and Manski (2006, 2009, 2010, 2011) report findings on various classes of policy choices. In my work to date, I have warned against analytical practices that promote incredible certitude. I have not, however, sought to classify these practices and consider them in totality. I do so here. My hope is to move future policy analysis away from incredible certitude and towards honest portrayal of partial knowledge. To begin, I distinguish the logic and the credibility of empirical research (Section 1) and cite arguments made for certitude (Section 2). I then develop a typology of practices that contribute to incredible certitude. I call these practices conventional certitude (Section 3), dueling certitudes (Section 4), conflating science and advocacy (Section 5), wishful extrapolation (Section 6), illogical certitude (Section 7), and media overreach (Section 8). To conclude I express my vision for credible policy analysis and raise some open questions (Section 9).